{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "FU2iDWOMEnYK"
      },
      "source": [
        "# Fish Diffusion\n",
        "<div style=\"display: flex; justify-content: center;\">\n",
        "<img alt=\"LOGO\" src=\"https://cdn.jsdelivr.net/gh/fishaudio/fish-diffusion@main/images/logo_512x512.png\" width=\"256\" height=\"256\" />\n",
        "</div>\n",
        "\n",
        "<style>\n",
        "  a {\n",
        "    margin-right: 16px;\n",
        "  }\n",
        "  div {\n",
        "    margin-top: 16px;\n",
        "  }\n",
        "</style>\n",
        "<div style=\"display: flex; justify-content: center; margin-bottom: 20px; \">\n",
        "<a href=\"https://discord.gg/wbYSRBrW2E\">\n",
        "<img alt=\"Discord\" src=\"https://img.shields.io/discord/1044927142900809739?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square\"/>\n",
        "</a>\n",
        "\n",
        "<a href=\"https://huggingface.co/spaces/fishaudio/fish-diffusion\">\n",
        "<img alt=\"Hugging Face\" src=\"https://img.shields.io/badge/🤗%20Spaces-HiFiSinger-blue.svg?style=flat-square\"/>\n",
        "</a>\n",
        "\n",
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/fishaudio/fish-diffusion/blob/main/notebooks/train.ipynb\">\n",
        "<img alt=\"Open In Colab\" src=\"https://img.shields.io/static/v1?label=Colab&message=Notebook&color=F9AB00&logo=googlecolab&style=flat-square\"/>\n",
        "</a>\n",
        "</div>\n",
        "\n",
        "</div>"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "F-Ym3LgeEnYM"
      },
      "source": [
        "## Terms of Use for Fish Diffusion\n",
        "\n",
        "1. Obtaining Authorization and Intellectual Property Infringement: The user is solely accountable for acquiring the necessary authorization for any datasets utilized in their training process and assumes full responsibility for any infringement issues arising from the utilization of the input source. Fish Diffusion and its developers disclaim all responsibility for any complications that may emerge due to the utilization of unauthorized datasets.\n",
        "\n",
        "2. Proper Attribution: Any derivative works based on Fish Diffusion must explicitly acknowledge the project and its license. In the event of distributing Fish Diffusion's code or disseminating results generated by this project, the user is obliged to cite the original author and source code (Fish Diffusion).\n",
        "\n",
        "3. Audiovisual Content and AI-generated Disclosure: All derivative works created using Fish Diffusion, including audio or video materials, must explicitly acknowledge the utilization of the Fish Diffusion project and declare that the content is AI-generated. If incorporating videos or audio published by third parties, the original links must be furnished.\n",
        "\n",
        "4. Agreement to Terms: By persisting in the use of Fish Diffusion, the user unequivocally consents to the terms and conditions delineated in this document. Neither Fish Diffusion nor its developers shall be held liable for any subsequent difficulties that may transpire."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "91Gn92gkHCgg"
      },
      "outputs": [],
      "source": [
        "#@title ## Agreement\n",
        "i_agree_the_terms_above = False #@param {type:\"boolean\"}\n",
        "\n",
        "if i_agree_the_terms_above is False:\n",
        "  raise Exception(\"You need to agree with the terms to continue.\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Ru3D89dQEnYN"
      },
      "source": [
        "## Environment Setup\n",
        "\n",
        "\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "V2q6Ze-HJNUZ"
      },
      "source": [
        "### Check GPU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "D5dkQ3JPIi49"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "import torch\n",
        "cuda_available = torch.cuda.is_available()\n",
        "if cuda_available is False:\n",
        "  raise Exception(\"CUDA is not available, please change instance type.\")\n",
        "\n",
        "for i in range(torch.cuda.device_count()):\n",
        "  print(f\"GPU {i}: {torch.cuda.get_device_name(i)} detected.\")\n",
        "\n",
        "print(\"-\" * 20)\n",
        "\n",
        "!nvidia-smi"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "jKQ1UNhoEnYN"
      },
      "source": [
        "### Configure Environment\n",
        "You may skip this section if you have correct environment (fish_diffusion) installed"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "jem4hACoQGsM"
      },
      "source": [
        "#### Install Conda"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "OPq_YPeGEnYN"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "%%bash\n",
        "mkdir /content/env\n",
        "MINICONDA_INSTALLER_SCRIPT=Miniconda3-py310_23.1.0-1-Linux-x86_64.sh\n",
        "MINICONDA_PREFIX=/content/env\n",
        "wget -q --show-progress https://repo.continuum.io/miniconda/$MINICONDA_INSTALLER_SCRIPT\n",
        "chmod +x $MINICONDA_INSTALLER_SCRIPT\n",
        "./$MINICONDA_INSTALLER_SCRIPT -b -f -p $MINICONDA_PREFIX"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "lIHbQhyGEnYO"
      },
      "source": [
        "#### Create Conda Environment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "cDUczyPDEnYO"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "%%bash\n",
        "source /content/env/bin/activate\n",
        "conda create -n fish_diffusion python=3.10 -y"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "m8Qnp231OQWz"
      },
      "source": [
        "#### Clone Repo"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "t4xQO_XkOQf_"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "import os\n",
        "\n",
        "if os.path.exists(\"/content/fish-diffusion\"):\n",
        "  print(\"The repo alerady exists, skipping\")\n",
        "else:\n",
        "  !git clone https://github.com/fishaudio/fish-diffusion /content/fish-diffusion\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ylBUnrn2EnYO"
      },
      "source": [
        "#### Install Dependencies\n",
        "Note, this error message is fine:  \n",
        "`Authorization error accessing https://download.pytorch.org/whl/cu118/wheel/`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "h4cdg5g2EnYP"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "%cd /content/fish-diffusion\n",
        "!source /content/env/bin/activate;\\\n",
        "conda activate fish_diffusion;\\\n",
        "curl -sSL https://raw.githubusercontent.com/pdm-project/pdm/main/install-pdm.py | python3 -;\\\n",
        "/root/.local/bin/pdm sync;"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "OVPBVq6CEnYP"
      },
      "source": [
        "## Vocoder preparation\n",
        "This section is used to prepare NSF-HiFiGAN for Diffusion (DiffSVC) training. It's optional in HiFiSinger."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "ayigcugjEnYP"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "%cd /content/fish-diffusion\n",
        "!source /content/env/bin/activate;\\\n",
        "conda activate fish_diffusion;\\\n",
        "python tools/download_nsf_hifigan.py --agree-license"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "CaekLP2WEnYP"
      },
      "source": [
        "## Dataset Preparation\n",
        "Currently, only single speaker-training is supported in this notebook.  \n",
        "\n",
        "### Option 1\n",
        "**You need to split your audios to 5-10 seconds segements before uploading them!!!**. \n",
        "To solve disk IO limitation, you need to upload your dataset in zip format with the following structure:\n",
        "\n",
        "```shell\n",
        "[ZIP ROOT]\n",
        "├───train\n",
        "│   ├───xxx1-xxx1.wav\n",
        "│   ├───...\n",
        "│   └───Lxx-0xx8.wav\n",
        "└───valid\n",
        "    ├───xx2-0xxx2.wav\n",
        "    ├───...\n",
        "    └───xxx7-xxx007.wav\n",
        "```\n",
        "\n",
        "You only need to pick 5-10 samples to the valid folder.  \n",
        "\n",
        "### Option 2\n",
        "If you want the program to do split and pick them automatically, only upload a folder of wavs:\n",
        "\n",
        "```shell\n",
        "[ZIP ROOT]\n",
        "├───xxx1-xxx1.wav\n",
        "├───...\n",
        "└───Lxx-0xx8.wav\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "RLxf2lB7RzAl"
      },
      "outputs": [],
      "source": [
        "#@title Preprocess Dataset\n",
        "\n",
        "import os\n",
        "import zipfile\n",
        "\n",
        "#@markdown Do you want the program to download zip from gdrive automatically, instead of upload it by yourself?\n",
        "download_from_gdrive = True #@param {type:\"boolean\"}\n",
        "#@markdown The path of google drive or relative to the `/content`\n",
        "dataset_path = \"Fish Diffusion/opencpop-wavs.zip\" #@param {type:\"string\"}\n",
        "#@markdown Do you want the program to automatically slice audio into segments?\n",
        "auto_slice = False #@param {type:\"boolean\"}\n",
        "#@markdown Do you want the program to automatically split train and valid set?\n",
        "auto_pick = True #@param {type:\"boolean\"}\n",
        "auto_pick_num = 5 #@param {type:\"number\"}\n",
        "\n",
        "if download_from_gdrive:\n",
        "  from google.colab import drive\n",
        "  drive.mount('/content/drive/')\n",
        "  zip_path = os.path.join(\"/content/drive/MyDrive\", dataset_path)\n",
        "else:\n",
        "  zip_path = os.path.join(\"/content\", dataset_path)\n",
        "\n",
        "z = zipfile.ZipFile(zip_path)\n",
        "files = list(z.namelist())\n",
        "has_train_folder = any(i.startswith(\"train\") for i in files)\n",
        "has_valid_folder = any(i.startswith(\"valid\") for i in files)\n",
        "has_folder = any(\"/\" in i for i in files)\n",
        "\n",
        "print(f\"Has train folder: {has_train_folder}\")\n",
        "print(f\"Has valid folder: {has_valid_folder}\")\n",
        "print(f\"Has folder: {has_folder}\")\n",
        "print(\"-\" * 20)\n",
        "\n",
        "!rm -rf /content/fish-diffusion/dataset\n",
        "\n",
        "if has_train_folder != has_valid_folder:\n",
        "  print(\"Your dataset structure is incorrect, you should have either both train and valid or none of them.\")\n",
        "elif not (has_train_folder and has_valid_folder) and has_folder:\n",
        "  print(\"Your dataset structure is incorrect, you shouldn't have any folders in your zip if you don't have both train and valid.\")\n",
        "elif has_folder and not (auto_slice is False and auto_pick is False):\n",
        "  print(\"Auto split and auto pick are not available when subfolders exist\")\n",
        "elif has_train_folder and has_valid_folder:\n",
        "  os.makedirs(\"/content/fish-diffusion/dataset\", exist_ok=True)\n",
        "  !unzip -q \"{zip_path}\" -d /content/fish-diffusion/dataset\n",
        "  print(\"OK\")\n",
        "else:\n",
        "  train_path = \"/content/fish-diffusion/dataset/train\"\n",
        "  valid_path = \"/content/fish-diffusion/dataset/valid\"\n",
        "  os.makedirs(train_path, exist_ok=True)\n",
        "  os.makedirs(valid_path, exist_ok=True)\n",
        "\n",
        "  if auto_slice:\n",
        "    print(\"Unzipping\")\n",
        "    raw_path = \"/content/fish-diffusion/dataset/raw\"\n",
        "    os.makedirs(raw_path, exist_ok=True)\n",
        "    !unzip -q \"{zip_path}\" -d \"{raw_path}\" \n",
        "    print(\"Unzip completed\")\n",
        "\n",
        "    # Call slicer\n",
        "    !/content/env/envs/fish_diffusion/bin/fap slice-audio \\\n",
        "      \"{raw_path}\" \"{train_path}\" --top-db 50 --num-workers 4\n",
        "    print(\"Audo sliced\")\n",
        "  else:\n",
        "    print(\"Unzipping\")\n",
        "    !unzip -q \"{zip_path}\" -d \"{train_path}\"\n",
        "    print(\"Unzip completed\")\n",
        "\n",
        "  !/content/env/envs/fish_diffusion/bin/python \\\n",
        "    /content/fish-diffusion/tools/preprocessing/random_move.py \\\n",
        "    \"{train_path}\" \"{valid_path}\" \"{auto_pick_num}\"\n",
        "\n",
        "  print(\"Copied 5 random files to valid folder\")\n",
        "  print(\"OK\")\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "_j3v-eS_g16g"
      },
      "source": [
        "## Choose Your Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "6XhBrmgIg_OP"
      },
      "outputs": [],
      "source": [
        "#@title Model Config\n",
        "#@markdown It's strongly recommand to use a pertrained model in colab\n",
        "pretrained = True #@param {type:\"boolean\"}\n",
        "pretrained_profile ='hifisinger-v2.1.0'#@param ['hifisinger-v2.1.0', 'diffusion-v2.0.0']\n",
        "\n",
        "PROFILES = {\n",
        "  \"hifisinger-v2.1.0\": {\n",
        "    \"arch\": \"hifisinger\",\n",
        "    \"config\": \"https://github.com/fishaudio/fish-diffusion/releases/download/v2.1.0/svc_hifisinger_finetune.py\",\n",
        "    \"model\": \"https://github.com/fishaudio/fish-diffusion/releases/download/v2.1.0/hifisinger-pretrained-20230329-540k.ckpt\"\n",
        "  },\n",
        "  \"diffusion-v2.0.0\": {\n",
        "    \"arch\": \"diffusion\",\n",
        "    \"config\": \"https://github.com/fishaudio/fish-diffusion/releases/download/v2.0.0/svc_content_vec_finetune.py\",\n",
        "    \"model\": \"https://github.com/fishaudio/fish-diffusion/releases/download/v2.0.0/content-vec-pretrained-v1.ckpt\"\n",
        "  }\n",
        "}\n",
        "\n",
        "#@markdown ---\n",
        "#@markdown Or bring your own config if you don't want to use pretrained\n",
        "arch = 'hifisinger'#@param ['diffusion', 'hifisinger']\n",
        "config_path = ''#@param{type:\"string\"}\n",
        "#@markdown Leaving the following field empty will disable pretrain\n",
        "pretrained_model_path = ''#@param{type:\"string\"}\n",
        "\n",
        "if pretrained:\n",
        "  profile = PROFILES[pretrained_profile]\n",
        "  arch = profile[\"arch\"]\n",
        "  config_path = f\"configs/svc_{profile['arch']}_finetune.py\"\n",
        "  pretrained_model_path = f\"checkpoints/pretrained_{profile['arch']}.ckpt\"\n",
        "\n",
        "  !wget -q --show-progress \"{profile['config']}\" -O \"/content/fish-diffusion/{config_path}\"\n",
        "  !wget -q --show-progress \"{profile['model']}\" -O \"/content/fish-diffusion/{pretrained_model_path}\"\n",
        "\n",
        "  print(f\"Your config is saved to {config_path}\")\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "vUcWIbYRk-pV"
      },
      "source": [
        "> The project is under active development, please backup your config file  \n",
        "> The project is under active development, please backup your config file  \n",
        "> The project is under active development, please backup your config file  "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "2E3WJgFAEnYQ"
      },
      "source": [
        "### Extract Features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "-MLHFk6-EnYQ"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "!source /content/env/bin/activate;\\\n",
        "conda activate fish_diffusion;\\\n",
        "python tools/preprocessing/extract_features.py --config \"{config_path}\" --path dataset/valid --clean --no-augmentation;\\\n",
        "python tools/preprocessing/extract_features.py --config \"{config_path}\" --path dataset/train --clean --num-workers 4"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "O_Y8c1c-EnYQ"
      },
      "source": [
        "## Final Steps"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "-t8CKJPFEnYQ"
      },
      "outputs": [],
      "source": [
        "#@title Training\n",
        "#@markdown You may want to resume from your previous checkpoint\n",
        "resume = False #@param {type:\"boolean\"}\n",
        "resume_model_path = ''#@param{type:\"string\"}\n",
        "\n",
        "logger = 'tensorboard' #@param ['wandb', 'tensorboard']\n",
        "#@markdown You may continue your wandb experiment by providing the experiment id\n",
        "resume_id = ''#@param{type:\"string\"}\n",
        "#@markdown Point to where you want to save models & logs in GDrive\n",
        "dest_path = '/content/drive/MyDrive/FishSVC/'#@param{type:\"string\"}\n",
        "\n",
        "args = \"\"\n",
        "if pretrained_model_path:\n",
        "  args += f\"--pretrain {pretrained_model_path} \"\n",
        "\n",
        "if logger == \"tensorboard\":\n",
        "  args += f\"--tensorboard \"\n",
        "  %load_ext tensorboard\n",
        "  %tensorboard --logdir .\n",
        "\n",
        "if resume:\n",
        "  args += f\"--resume {resume_model_path} \"\n",
        "  if logger == \"wandb\" and resume_id:\n",
        "    args += f\"--resume-id {resume_id} \"\n",
        "if dest_path:\n",
        "   args += f\"--dest-path {dest_path} \"\n",
        "\n",
        "!source /content/env/bin/activate;\\\n",
        "conda activate fish_diffusion;\\\n",
        "python \"tools/{arch}/colab_train.py\" --config \"{config_path}\" {args}"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Kd-b14dmJmPf"
      },
      "source": [
        "After training, you can find your checkpoints in `/content/fish-diffusion/logs/HiFiSVC/[VERSION]/checkpoints`, make sure to backup them."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Z0n_Vq9fo_PJ"
      },
      "source": [
        "### Inference"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "k78fLXQpJ3qn"
      },
      "outputs": [],
      "source": [
        "#@title Gradio UI\n",
        "#@markdown You need to run the model_config block before running this one.   \n",
        "\n",
        "#@markdown The checkpoint path you want to use. You can use a folder if you want the code to atutomatically find the latest one\n",
        "checkpoint_path = \"/content/fish-diffusion/logs/HiFiSVC/version_0/checkpoints\" #@param {type:\"string\"}\n",
        "\n",
        "import yaml\n",
        "\n",
        "if arch == \"hifisinger\":\n",
        "  gradio_config = {\n",
        "    \"readme\": \"# Fish Diffusion - HiFiSinger Demo 🎤\\nGitHub Repo: [fishaudio/fish-diffusion](https://github.com/fishaudio/fish-diffusion) \\nTo share a new model, please check out the [Share Your Model](https://huggingface.co/spaces/fishaudio/fish-diffusion/discussions/2) discussion.\\n\",\n",
        "    \"max_mixing_speakers\": 3,\n",
        "    \"models\": [\n",
        "      {\n",
        "        \"name\": \"demo\",\n",
        "        \"config\": config_path,\n",
        "        \"checkpoint\": checkpoint_path,\n",
        "        \"readme\": \"This model is pretrained on the Opencpop and M4Singer dataset and fintuned on your dataset.\",\n",
        "      }\n",
        "    ]\n",
        "  }\n",
        "\n",
        "  with open(\"gradio-config.yaml\", \"w\") as f:\n",
        "    yaml.dump(gradio_config, f)\n",
        "\n",
        "  !source /content/env/bin/activate;\\\n",
        "    conda activate fish_diffusion;\\\n",
        "    python -c 'import gradio;gradio.close_all()';\\\n",
        "    python /content/fish-diffusion/tools/hifisinger/gradio_ui.py \\\n",
        "    --config gradio-config.yaml --share\n",
        "else:\n",
        "  !source /content/env/bin/activate;\\\n",
        "    conda activate fish_diffusion;\\\n",
        "    python tools/diffusion/inference.py \\\n",
        "    --config \"{config_path}\" --checkpoint \"{checkpoint_path}\" \\\n",
        "    --gradio --gradio_share"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "XnpUjfBdYCkC"
      },
      "source": [
        "#### Command Line\n",
        "This is not recommanded for beginners, but it gives you more flexibility."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lKTYACAPYAUM"
      },
      "outputs": [],
      "source": [
        "!source /content/env/bin/activate;\\\n",
        "    conda activate fish_diffusion;\\\n",
        "    python \"tools/{arch}/inference.py\" \\\n",
        "    --config \"{config_path}\" --checkpoint \"{checkpoint_path}\" \\\n",
        "    --input \"input.wav\" \\\n",
        "    --output \"output.wav\" \\\n",
        "    --speaker 0 --pitch_adjust 0"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "gpuClass": "premium",
    "kernelspec": {
      "display_name": "fish-audio",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.9"
    },
    "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
